Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[LoRA] Adds example on text2image fine-tuning with LoRA #2031

Merged
merged 16 commits into from
Jan 23, 2023
Merged

[LoRA] Adds example on text2image fine-tuning with LoRA #2031

merged 16 commits into from
Jan 23, 2023

Conversation

sayakpaul
Copy link
Member

@sayakpaul sayakpaul commented Jan 18, 2023 •
edited
Loading

Very much copied from @patrickvonplaten's awesome work with #1884

Things seem to be working on both T4 and V100.

My command:

export MODEL_NAME="CompVis/stable-diffusion-v1-4"
export DATASET_NAME="lambdalabs/pokemon-blip-captions"

accelerate launch --mixed_precision="fp16" \
  train_text_to_image_lora.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --dataset_name=$DATASET_NAME --caption_column="text" \
  --resolution=512 --random_flip \
  --train_batch_size=1 \
  --num_train_epochs=100 --checkpointing_steps=5000 \
  --learning_rate=1e-04 --lr_scheduler="constant" --lr_warmup_steps=0 \
  --seed=42 \
  --enable_xformers_memory_efficient_attention \
  --validation_prompt="cute Sundar Pichai creature" --report_to="wandb" \
  --output_dir="sd-model-finetuned-lora-t4" \
  --push_to_hub && sudo shutdown now

The final weights will be pushed to https://huggingface.co/sayakpaul/sd-model-finetuned-lora-v100 and an experimentation run is available here: https://wandb.ai/sayakpaul/text2image-fine-tune/runs/3iiol807 (currently running). Once these are done, I will update the appropriate sections in the README.

@sayakpaul sayakpaul self-assigned this Jan 18, 2023
@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Jan 18, 2023 •
edited
Loading

The documentation is not available anymore as the PR was closed or merged.

@sayakpaul sayakpaul changed the title example on fine-tuning with LoRA. [LoRA] Adds example on text2image fine-tuning with LoRA Jan 18, 2023
@sayakpaul sayakpaul marked this pull request as ready for review January 18, 2023 18:29
@sayakpaul sayakpaul mentioned this pull request Jan 18, 2023
Closed
patil-suraj
patil-suraj approved these changes Jan 19, 2023
View reviewed changes
Copy link
Contributor

@patil-suraj patil-suraj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for working on this! Just left some nits. And we should also add the model card creation function in this script. cf #2032

examples/text_to_image/README.md Outdated Show resolved Hide resolved
examples/text_to_image/README.md Outdated Show resolved Hide resolved
examples/text_to_image/README.md Outdated Show resolved Hide resolved
examples/text_to_image/train_text_to_image_lora.py Show resolved Hide resolved
examples/text_to_image/train_text_to_image_lora.py Outdated Show resolved Hide resolved
examples/text_to_image/train_text_to_image_lora.py Outdated Show resolved Hide resolved
examples/text_to_image/train_text_to_image_lora.py Outdated Show resolved Hide resolved
examples/text_to_image/train_text_to_image_lora.py Outdated
state_dict = lora_layers.state_dict()
lora_layers.load_state_dict(state_dict)

accelerator.register_for_checkpointing(lora_layers)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think we actually don't need it. If we are passing lora_layers to accelerate.prepare and if it's nn.Module it will be automatically checkpointed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's in accelerator.prepare so we can remove it here

examples/text_to_image/train_text_to_image_lora.py Show resolved Hide resolved
examples/text_to_image/train_text_to_image_lora.py Outdated Show resolved Hide resolved
patrickvonplaten
patrickvonplaten reviewed Jan 19, 2023
View reviewed changes
Copy link
Contributor

@patrickvonplaten patrickvonplaten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Super nice! Great that you made it work so quickly. Think only a couple of things are left to do :-)

sayakpaul and others added 6 commits January 19, 2023 23:07
@sayakpaul @patil-suraj @patrickvonplaten
Apply suggestions from code review
5260ec6 
Co-authored-by: Suraj Patil <surajp815@gmail.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
@sayakpaul
Merge branch 'add/lora-text-to-image' of https://github.com/huggingfa…
fbb2935 
…ce/diffusers into add/lora-text-to-image
@sayakpaul @patrickvonplaten
apply suggestions for PR review.
14dd3ea 
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
@sayakpaul
apply make style and make quality.
73c16f8
@sayakpaul
chore: remove mention of dreambooth from text2image.
8027f5f
@sayakpaul
add: weight path and wandb run link.
dc3b615
@sayakpaul
Copy link
Member Author

@patil-suraj @patrickvonplaten I think we're good to go :)

patrickvonplaten
patrickvonplaten reviewed Jan 23, 2023
View reviewed changes
examples/text_to_image/README.md

**___Note: When using LoRA we can use a much higher learning rate compared to non-LoRA fine-tuning. Here we use *1e-4* instead of the usual *1e-5*. Also, by using LoRA, it's possible to run `train_text_to_image_lora.py` in consumer GPUs like T4 or V100.**

The final LoRA embedding weights have been uploaded to [sayakpaul/sd-model-finetuned-lora-t4](https://huggingface.co/sayakpaul/sd-model-finetuned-lora-t4). **___Note: [The final weights](https://huggingface.co/sayakpaul/sd-model-finetuned-lora-t4/blob/main/pytorch_lora_weights.bin) are only 3 MB in size, which is orders of magnitudes smaller than the original model.**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice!

@patrickvonplaten
Copy link
Contributor

Thanks a lot for your work here @sayakpaul - let's try to advertise it nicely now :-)

@patrickvonplaten patrickvonplaten merged commit ffb3a26 into main Jan 23, 2023
@patrickvonplaten patrickvonplaten deleted the add/lora-text-to-image branch January 23, 2023 07:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@patrickvonplaten patrickvonplaten patrickvonplaten left review comments

@patil-suraj patil-suraj Awaiting requested review from patil-suraj

Assignees

@sayakpaul sayakpaul

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@sayakpaul @HuggingFaceDocBuilderDev @patrickvonplaten @patil-suraj